Papers with retrieval-augmented language models
Unveiling the Magic: Investigating Attention Distillation in Retrieval-Augmented Generation (2024.naacl-short)
Copied to clipboard
| Challenge: | Retrieval-augmented generation framework addresses the limitations of large language models by enabling real-time knowledge updates for more accurate answers. |
| Approach: | They propose to use attention distillation to improve retrieval-augmented language models' learning performance by identifying key factors influencing their workflow and proposing indicators for optimizing models’ training methods and avoiding ineffective training. |
| Outcome: | The proposed framework improves the learning performance of large language models in the training phase but also reduces the impact of ineffective training. |
Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing retrieval-augmented language models assume query relevance and irrelevance as dichotomy . existing models are highly brittle to the presence of conflicting information in both the fine-tuning and in-context few-shot learning scenarios. |
| Approach: | They propose methods for handling knowledge conflicts by fine-tuning a discriminator or prompting it to elicit its discriminative capability. |
| Outcome: | The proposed approaches significantly enhance model robustness on open-domain QA. |
GrowOVER: How Can LLMs Adapt to Growing Real-World Knowledge? (2024.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge-based datasets are outdated due to the rapid evolution of knowledge. |
| Approach: | They propose a retrieval-interactive language model framework that evaluates and reflects on its answers for further re-retrieval. |
| Outcome: | The proposed framework performs comparably to or surpasses continuously trained language models. |
Long-Tail Crisis in Nearest Neighbor Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Prior studies have shown that kNN-LM can retrieve long-tail contexts, leaving the model’s performance underexplored in estimating the probabilities of long-tailed target tokens. |
| Approach: | They investigate the behavior of kNN-LM on low-frequency tokens, examining prediction probability, retrieval accuracy, and token distribution in the datastore. |
| Outcome: | The proposed model improves the perplexity of given text by directly accessing a large datastore built from any text data during inference. |
Reimagining Retrieval Augmented Language Models for Answering Queries (2023.findings-acl)
Copied to clipboard
Wang-Chiew Tan, Yuliang Li, Pedro Rodriguez, Richard James, Xi Victoria Lin, Alon Halevy, Wen-tau Yih
| Challenge: | Large language models (LLMs) are expensive to train, deploy, and maintain, both financially and in terms of environmental impact. |
| Approach: | They present a reality check on large language models and compare their predictions to retrieval-augmented language models. |
| Outcome: | The proposed models fare better on question answering tasks and have become the foundation of impressive demos like Chat-GPT. |
AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current language models and retrieval-augmented LMs are limited in their ability to perform tasks on the web. |
| Approach: | They propose a benchmark to evaluate language agents built on top of language models . they propose 'AssistantBench' which includes 214 tasks that can be automatically evaluated . |
| Outcome: | The proposed agent outperforms existing agents in a new benchmark for language agents on the web. |
Dense X Retrieval: What Retrieval Granularity Should We Use? (2024.emnlp-main)
Copied to clipboard
| Challenge: | a learned dense retrieval model is often overlooked when using a corpus for inference, resulting in a design choice of retrieval unit . granularity of retrievals is important for both retrieval and downstream tasks . |
| Approach: | They propose a retrieval unit for dense retrieval that uses propositions to index corpus . propositions are defined as atomic expressions within text, each encapsulating a distinct factoid . |
| Outcome: | The proposed retrieval unit outperforms passage-level units on retrieval and downstream tasks. |
kNN-LM Does Not Improve Open-ended Text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Interpolation-based retrieval-augmented language models (LMs) are a subtype of retrieval augmented language model that computes the probability of the next token by interpolating between the softmax distribution of the original LM and a token distribution formed by retrieving over an external datastore. |
| Approach: | They propose to interpolate the predicted distribution of the next word with a distribution formed from the most relevant retrievals for a given prefix. |
| Outcome: | The proposed methods do not exhibit improvements in open-ended generation quality, as measured by automatic evaluation metrics and human evaluations. |
RADCoT: Retrieval-Augmented Distillation to Specialization Models for Generating Chain-of-Thoughts in Query Expansion (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated superior performance to that of small language models in information retrieval for various subtasks including dense retrieval, reranking, query expansion, and pseudo-document generation. |
| Approach: | They propose a retrieval-augmented model specialization that distills the capability of LLMs to generate the chain-of-thoughts (CoT) for query expansion into a RADCoT. |
| Outcome: | The proposed model can generate the chain-of-thoughts (CoT) for query expansion, reducing the burden of internalizing and retaining world knowledge in model parameters. |
Connecting the Knowledge Dots: Retrieval-augmented Knowledge Connection for Commonsense Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that large language models exhibit a limited understanding of commonsense reasoning due to the necessity of implicit knowledge that is rarely expressed in text. |
| Approach: | They propose a retrieval-augmented knowledge connection framework that transforms indirectly relevant documents into a direct explanation to answer a given question. |
| Outcome: | The proposed framework outperforms state-of-the-art (SOTA) benchmarks and achieves +2.0% and +4.6% average accuracy on in-domain (ID) and out-of domain (OOD) benchmark. |